Migrate conversation language field default to 'en' - #11349

Defer Firebase initialization until migration execution

Move Firebase initialization and firestore.client() into main() (and inject the client into the helpers). The required python backend/scripts/scan_import_time_side_effects.py check reports this new top-level firebase_admin.initialize_app call, so the commit cannot pass preflight; it also prevents importing the migration helpers without initializing credentialed infrastructure.

AGENTS.md reference: backend/AGENTS.md:L235-L235

Useful? React with 👍 / 👎.

Addressed in 8a6c13d: Firebase init and client construction are fully deferred; the migration calls database._client.get_firestore_client() only inside execution paths, and scan_import_time_side_effects.py reports 0 violations.

Fail the migration when any user update fails

If a Firestore read or batch commit fails for even one user, this handler only logs the exception and the process still prints Done and exits successfully. A transient failure or concurrent document deletion can therefore leave part of the database unmigrated while deployment automation or an operator records the migration as successful; track failures and return a nonzero exit status after all futures finish.

Useful? React with 👍 / 👎.

Addressed in 8a6c13d: worker failures are counted and the process exits non-zero (sys.exit(1)), so automation cannot record a partial run as successful. Covered by test_worker_failure_exits_nonzero.

Add regression coverage for the new language defaults

Add tests proving that omitted language values default to en for all three changed conversation models, and that the migration updates missing/empty values while preserving existing languages and exposing write failures. This commit changes production defaults and adds a data migration without modifying any test, contrary to the repository's required behavior-change coverage.

AGENTS.md reference: AGENTS.md:L28-L28

Useful? React with 👍 / 👎.

Addressed: test_conversation_language_default.py proves omitted language defaults to 'en' on all three models with explicit values preserved; test_migrate_language_to_en.py covers missing/empty backfill, existing-language preservation, dry-run, and failure exit.

Preserve the detected language when creating sync conversations

The sync pipeline derives language from the user's explicit preference or STT detection, but constructs CreateConversation without passing it and supplies it only as the separate processing argument (utils/sync/pipeline.py:1075,1149-1162). With this new default, _get_conversation_obj serializes such Spanish or other non-English conversations as language='en'; subsequent reprocessing uses the persisted conversation.language, so summaries and derived data can be regenerated in English. Pass the resolved language into the model rather than allowing the default to replace a known value.

Useful? React with 👍 / 👎.

Addressed in 02fb397: process_segment now passes the resolved language into CreateConversation, with regression test test_detected_language_is_stored_on_new_conversation (French-detected sync persists 'fr').

Stream projected conversation fields during the backfill

For users with long histories or large encrypted transcript documents, list(conversations_ref.stream()) downloads every field and retains the user's entire conversation history in memory; with 64 users processed concurrently, the production-wide migration can exhaust memory and transfer far more Firestore data than needed before it reaches the updates. Iterate the stream directly and project only language so memory remains bounded and transcript blobs are not fetched.

Useful? React with 👍 / 👎.

Addressed in 8a6c13d: conversations are streamed with .select(['language']) only, no list() materialization; users are paged by document id so memory stays bounded.

P1: With CreateConversation.language now defaulting to 'en', any caller that constructs CreateConversation without explicitly passing the detected/preferred language (e.g. the sync pipeline, which resolves language separately) will silently persist language='en' even for non-English conversations. Since conversation.language is used for later reprocessing, this can cause summaries and derived data to be regenerated in English for non-English conversations. Ensure all callers pass the resolved language into the model instead of relying on the new default.

Check if this issue is valid — if so, understand the root cause and fix it. At backend/models/conversation.py, line 289:

<comment>With `CreateConversation.language` now defaulting to `'en'`, any caller that constructs `CreateConversation` without explicitly passing the detected/preferred language (e.g. the sync pipeline, which resolves language separately) will silently persist `language='en'` even for non-English conversations. Since `conversation.language` is used for later reprocessing, this can cause summaries and derived data to be regenerated in English for non-English conversations. Ensure all callers pass the resolved language into the model instead of relying on the new default.</comment>

<file context>
@@ -286,7 +286,7 @@ class CreateConversation(BaseModel):
 
     source: ConversationSource = ConversationSource.omi
-    language: Optional[str] = None
+    language: Optional[str] = 'en'
 
     processing_conversation_id: Optional[str] = None
</file context>

Covered by the fix for 3746820357 (02fb397): the sync pipeline threads the detected/preferred language into CreateConversation and a non-English sync regression test asserts it.

Use the shared Firestore client for migration credentials

When an operator uses the backend-supported SERVICE_ACCOUNT_JSON credential mode, credentials.ApplicationDefault() ignores that variable and the migration exits because it only accepts ADC via GOOGLE_APPLICATION_CREDENTIALS. Use database._client.get_firestore_client() so the migration honors the repository's credential preparation and configured Firestore target instead of requiring operators to translate supported credentials manually.

AGENTS.md reference: backend/AGENTS.md:L191-L193

Useful? React with 👍 / 👎.

Addressed in 8a6c13d: the migration uses database._client.get_firestore_client(), which routes through prepare_google_credentials and honors SERVICE_ACCOUNT_JSON.

Backfill Omi-source conversations too

Historical sync-created conversations are stored with source omi: process_segment defaults its source to ConversationSource.omi, and before this commit its CreateConversation call omitted the language. This filter therefore skips exactly those existing records that can lack the field, leaving their persisted language unset even though the migration reports success. Include omi in the backfill scope rather than limiting it to the legacy Friend source names.

Useful? React with 👍 / 👎.

Addressed: the migration applies to all conversation sources (no source filter), so omi-source sync conversations with missing language are backfilled too.

Restore the ratchet to the unchanged source count

Reset this entry to 2499 and remove the unrelated baseline raise. backend/utils/sync/pipeline.py is not modified by this commit and remains 2499 lines, so running check_product_file_line_count_ratchet.py against the commit reports both that the source file is absent from the diff and that the raised limit does not match its line count; this makes the required preflight contract fail.

AGENTS.md reference: AGENTS.md:L34-L35

Useful? React with 👍 / 👎.

The baseline was already restored to 2499 at eff8ac8; it is now raised to 2500 with a justification only because 02fb397 adds the language=language line to pipeline.py, and the ratchet check passes.

Report dry-run results as prospective updates

When the migration is invoked with --dry-run, process_user_conversations increments both counters without committing any writes, but this final message still reports that the conversations were "updated." An operator relying on the preview output as rollout evidence could therefore mistake a dry run for a completed migration; label these as conversations that would be updated when args.dry_run is true.

Useful? React with 👍 / 👎.

Addressed in 8a6c13d: dry-run output reports 'would be updated' instead of 'updated'; asserted in test_dry_run_reports_without_committing.

Pass a document reference to the pagination cursor

When the full migration reaches its second page, or whenever --start-after is supplied, this passes a raw string as the cursor value for order_by('__name__'). Firestore document-ID cursors require a document snapshot/reference (the repository's other paginated queries pass the last snapshot or collection.document(id)), so the production query rejects this cursor even though the fake test accepts it; the migration therefore cannot progress beyond the first page. Use the last snapshot or a document reference in the cursor.

Useful? React with 👍 / 👎.

Enforce --max-writes while work is running

For every invocation with --max-writes, total_writes remains zero throughout this entire scheduling loop because results are not collected until the subsequent as_completed loop. Consequently the limit never stops scheduling, and a staged run such as --max-writes 1000 can update the whole database. Results need to be consumed while scheduling, with the remaining write budget enforced before additional user work is allowed.

Useful? React with 👍 / 👎.

Do not resume after an out-of-order completion

When any worker fails, last_uid is merely the most recently completed successful future, not the greatest contiguous user ID processed before the failure. For example, if u1 fails and u2 completes afterward, the suggested --start-after u2 retry permanently skips u1 while the idempotent migration appears complete. Track a contiguous ordered checkpoint, explicitly retry failed IDs, or advise rerunning without this cursor.

Useful? React with 👍 / 👎.

Bound the number of submitted user futures

On the default unlimited full run, this submits one future for every user before collecting any result, and both the executor's unbounded queue and futures retain all of them. The user iterator's paged reads and eight-worker setting therefore do not bound memory; on the repository's 100K-plus user collection, the migration can consume substantial memory or be killed before processing finishes. Keep only a small bounded window of futures in flight and replenish it as each completes.

Useful? React with 👍 / 👎.

Enforce the write cap inside each worker

When a user has more missing-language conversations than the remaining --max-writes budget, this worker receives no budget and commits every matching conversation before returning; the coordinator checks the limit only afterward. Thus a staged --max-writes 1000 run can perform thousands of writes for one large user, plus writes from other in-flight workers. Fresh evidence on the current head is that the bounded scheduler still calls process_user_conversations without a remaining-write limit, so the earlier post-hoc cap issue remains.

Useful? React with 👍 / 👎.

Drain pending results before reporting success

With --max-writes and multiple workers, reaching the limit breaks out while other futures remain pending; exiting the executor context waits for those workers, so they can still commit, but their results and exceptions are never consumed. If one of those commits fails, failures remains zero and the migration logs Done and exits successfully despite a partial backfill. Fresh evidence on the current head is this new early break, which bypasses the failure-accounting path for all remaining futures.

Useful? React with 👍 / 👎.

Keep UserDefaults delivery off the main queue

When a background auth commit writes UserDefaults while the main thread is waiting on the session fence, queue: .main synchronously waits for the main thread and recreates the frozen sign-in deadlock fixed by 6467d6fc8. This backend-language commit reverses that fix; retain queue: nil and asynchronously hop to the main actor instead.

AGENTS.md reference: desktop/macos/AGENTS.md:L356-L359

Useful? React with 👍 / 👎.

Fail when a migration worker update fails

If any Firestore stream or batch commit raises, this handler logs the exception but still reaches Done and exits successfully, allowing automation to record a partially migrated database as complete. Although the earlier thread cites 8a6c13d5e6 as the fix, that commit is not present in this reviewed tree; these catch-and-continue lines are fresh evidence that failures still need to be accumulated and produce a nonzero exit.

Useful? React with 👍 / 👎.

Pass the resolved sync language into the create model

For a new sync conversation whose detected or explicitly selected language is non-English, process_segment still constructs CreateConversation without language, so this default makes _get_conversation_obj persist en even though processing received the resolved language separately. The prior thread's cited fix 02fb397539 is not an ancestor of this reviewed commit; the unchanged omission at backend/utils/sync/pipeline.py:1149-1158 is fresh evidence that the resolved language must be passed into this model.

Useful? React with 👍 / 👎.

Guard Crashlytics until Firebase initialization succeeds

If _init() throws before a default Firebase app exists—for example because flavor configuration is missing or invalid—the zone handler calls FirebaseCrashlytics.instance, which itself requires that Firebase app and throws a second error, masking the original startup failure and losing its diagnostics. Restore the Firebase.apps.isNotEmpty guard and a non-Firebase logging fallback rather than reverting the existing startup-error fix in this unrelated backend change.

AGENTS.md reference: AGENTS.md:L50-L52

Useful? React with 👍 / 👎.

Return 405 for idle MCP GET streams

Every MCP client that opens this GET now occupies a Cloud Run request slot in an infinite ping loop until disconnect or the one-hour request timeout, despite the endpoint carrying no protocol messages. This directly reverses the parent commit's production fix after roughly 1,200 concurrent parked streams exhausted container concurrency and pushed MCP tool-call latency into minutes; keep the stateless transport's 405 Method Not Allowed response instead.

AGENTS.md reference: AGENTS.md:L50-L52

Useful? React with 👍 / 👎.

Deduplicate concurrent transcription socket attempts

When the 15-second keep-alive tick fires while an earlier _initiateWebsocket call is still connecting, this implementation starts another identical attempt; the socket pool serializes but does not deduplicate them, so both can eventually open /v4/listen sessions and race the controller and UI state. Restore the per-configuration in-flight guard removed by this unrelated rollback while continuing to allow forced or differently configured attempts.

AGENTS.md reference: AGENTS.md:L50-L52

Useful? React with 👍 / 👎.

Re-emit device-ready for an existing Android BLE link

When Background Mode keeps the service and GATT connection alive after the Flutter engine dies, the next app launch routes its connect request to forceReconnect, which immediately returns for an already-connected peripheral and never emits Dart's only onDeviceReady signal. The app consequently waits through the connection timeout and reports the device disconnected despite the live link, disabling transcription until a manual disconnect; route this through the adopt-and-resync path restored by the reverted Android fix.

AGENTS.md reference: AGENTS.md:L50-L52

Useful? React with 👍 / 👎.

Preserve the production memory cursor secret binding

On the next production Cloud Run or GKE deployment, this overlay no longer injects MEMORY_V3_CURSOR_SECRET, even though /v1/knowledge-graph and signed canonical-memory cursors fail closed when the HMAC secret is absent. Production was already repaired to consume this secret, so deploying this tree strips the live binding and makes Mind Map and cursor-backed reads return missing_cursor_secret; restore the secret and version entries in every production serving target.

AGENTS.md reference: backend/AGENTS.md:L171-L173

Useful? React with 👍 / 👎.

Do not start foreground audio from location permission

For every mobile user who has granted location permission, opening Home now starts flutter_foreground_task even when no capture is active. On iOS this holds AVAudioSession and leaves the green microphone indicator and foreground service running while idle, producing the all-day battery drain fixed by b5b77f5fa; foreground audio ownership must instead follow an active capture with recent frames.

AGENTS.md reference: AGENTS.md:L50-L52

Useful? React with 👍 / 👎.